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We, the undersigned, hereby declare that 



1 . We are the inventors of the invention disclosed and claimed in the United 
States Patent Application having serial number 10/680,205, filed October 8, 2003 
("the '205 application"). 



2. The '205 application claims priority to Japanese Patent Application No. JP 
2002-295,429 filed in Japan on October 8, 2002. 



Examiner: Jayesh A. Patel 
Art Unit: 2624 
Confirmation No.: 7459 



1 



3. Exhibit A is a printout of an e-mail prepared by Mr. Satoru Adachi, one of 
the inventors of the '205 application, and sent to Mr. Thiow Keng Tan, another inventor 
of the '205 application. Although the date of the e-mail is redacted in Exhibit A, the e- 
mail was prepared and sent prior to October 3, 2002, the effective date of U.S. Patent 
Application Publication No. 2004-0066974 (Karczewicz et al.). 

4. Exhibit A reported the results of simulations conducted regarding the 
performance of our invention and evaluations of the simulation results made by Mr. 
Adachi. As described in Exhibit A, the simulations were conducted to implement 
"CAVLC on ABT", using three different methodologies, "Split," "Real" and "Interleave." 
The purpose of the simulations was to determine a methodology which could exhibit the 
best performance among the three in implementing CAVLC on ABT. 

5. In "Split," transform coefficients in a block having a size of, for example, 
8x8 are zig-zag scanned and arranged into a string of 64 coefficients (one dimensional), 
and then the string is divided at equal intervals into four strings, each having 16 
coefficients. 

6. In "Real," an 8x8 block having 64 transform coefficients is divided into four 
quadrants, each being a 4x4 sub-block, and the transform coefficients in each sub-block 
is zig-zag scanned and arranged into a string of 16 coefficients, yielding four strings 
each having 16 coefficients. 

7. In "interleave," which is our invention, 64 transform coefficients in an 8x8 
block are zig-zag scanned and interleaved into four strings each having 16 transform 
coefficients. 

8. We prepared simulation programs for the respective three methodologies 
and run the programs to implement CALVC on ABT on the three methodologies. 
Exhibit B is a printout of a part of the results of the simulations from "CAVLC 
performance on ABT coeff split.xls" mentioned in Exhibit A. 
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9. In Exhibit B, prior art is identified as "JM4.0d." The graphs in Exhibit B 
showed the performances (%) of the three methodologies improved over JM4.0dAs in 
relation to various compression levels (12, 16, 20, 24, 28, 32, 36, 40). Thus, a 
methodology which exhibited higher performance in the simulations manifested itself 
higher in the graphs. It was our observation that the overall simulation results favored 
our invention "Interleave" most among the three methodologies. 

1 0. Exhibit C is an excerpt from the simulation program to implement our 
invention, "Interleave," in the encoding process. Exhibit D is an excerpt from the 
simulation program to implement our invention in the decoding process. 

11. In Exhibit A, Mr. Adachi indicated that he had not yet implemented a 
decoder. He meant that he had not run a simulation program for a decoder as of the 
date of Exhibit A. However, as of the date of Exhibit A, he had confirmed the 
performance of our invention in the decoding process in the simulations for an encoder 
because an encoder necessarily implements the decoding process. 

12. Encouraged by the simulation results, we decided to propose our 
invention to Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG and file a patent 
application for the invention. Exhibit E is an e-mail prepared and sent to spg-visual by 
Mr. Adachi subsequent to the simulations, yet still prior to October 3, 2002. "spg-visual" 
was a representative mail address of the visual team in the signal processing group, to 
which Mr. Adachi belonged. The team leader of the visual team was then Mr. Minoru 
Etoh, another inventor of our invention. The team also included Mr. Sadaatsu Kato, 
another inventor of our invention. Our invention is noted in Exhibit E as "A method for 
applying context-adaptive variable-length coding to adaptive orthogonal transform size 
image encoding." 

1 3. In Exhibit E, Mr. Adachi indicated that he would complete drafting of a 
patent application for our invention as soon as he could. As promised, Mr. Adachi 
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worked on drafting the patent application every day from the date of Exhibit E through 
October 8, 2002, including the days from October 3, 2002 through October 8, 2002. 

14. Mr. Adachi knew that he had to file the patent application no later than 
October 8, 2002 and that very little time would be given to Japanese counsel to perfect 
his draft application before the filing thereof in the Japanese Patent Office. Therefore, 
Mr. Adachi spent a significant amount of time every day to try to prepare as perfect and 
comprehensive a draft application as possible so that Japanese counsel would need 
little time to prepare a formal patent application from his draft application. 

15. A byproduct of Mr. Adachi's efforts in preparing a draft application was a 
proposal to Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG. Exhibit F is the 
proposal to JVT prepared by Mr. Adachi and uploaded on a server of JVT on October 5, 
2002 for discussion at a meeting held in Switzerland on October 9-17, 2002. Exhibit F 
described our invention and we believe serves an indication of how much Mr. Adachi's 
draft application was completed as of October 5, 2002. 

16. It took Mr. Adachi three more days to complete his draft application after 
the proposal to JVT was uploaded. Exhibit G is the draft patent application prepared by 
Mr. Adachi which was completed on October 8, 2002 and sent to the Japanese counsel 
on the same day. 

17. Exhibit H is the drawings prepared by Mr. Adachi which were completed 
on October 8, 2002 for his draft application. 

18. The Japanese counsel prepared a formal patent application from Mr. 
Adachi's draft application and filed it in the Japanese Patent Office on October 8, 2002, 
which was later assigned Serial Number JP 2002-295,429. 

We hereby declare that all statements made herein are of our own knowledge 
and are true and that all statements made on information and belief are believed to be 
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true, and further that these statements are made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under 
Section 1001 of Title 18 of the Unites States code, and that such willful statements may 
jeopardize the validity of the application or any patent issued therefrom. 



Respectfully submitted, 



SatoruAdachi ✓ Date 





Minoru EtSh Date 

>Xl ImM^ l<As(yC z>f\ /mi* 




Date 



^/ f/ p-oro 



Thiow Keng Tan Date 
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Exhibit A 



> Date: HHHHHBB 

> To: <tktan> 

> From: ADACHI Satoru <adachi> 

> Subject: RE: FW: Qvt_vlc_adhoc] CAVLC for ABT 

> 

> Dear TK, 
> 

> Attached please find data for CAVLC on ABT. 
> 

> » CAVLC_ABT_addCIFaBBT.xls: 

> 

> Results of your implementation. I added, to your data, my results of GIF 

> sequences, tempete, bus, flowergarden in addition to mobile, paris. Data 

> of "JM4.0d" and "CAVLC all blk sz" is simply copied from Rears results, 

> as I did for mobile and paris. 
> 

> I also add my additional CIF results on your latest Excel file y ou just ^ 

> sent me. The revised one is jm40c_abt2_adachLttk_addCIFJ^B|.xls. 

> 

> » CAVLC performance on ABT coeff splitMHH.xls: 

> 

> Comparison of our implementation. We implemented "real" (division as Real 

> did, but it implemented ourselves), "split", and "interleave". We 

> introduced simple reordering and division of ABT coefficients, and not 

> introduce new table selection nor CBP. 
> 

> Please compare "CAVLC^BTjaddCIF^HB-XIs'' and "CAVLC performance 
ABT 

> coeff split ■■R.xls" each other. You can see that our "real" results 

> and "CAVLC all BLK sz", and, our "split" results and your "abt scan split 

> + NTS", have the same tendency respectively. I think it shows correctness 

> of our implementation to some extent. (To tell the truth, we did not 

> implement decoder yet! We will do it ASAP!) 
> 

> Although we need to carefully check the correctness of our 

> implementation, from the results, I think we can see some interesting findings. 
> 

> 1 . Rears approach does provide good improvement with no modification on 

> current CAVLC. 
> 

> 2. "split" does not work better than Real's. The possible gain provided 

> by new table selection seems not to recover this inefficiency. 
> 

> 3. "interleave" works best among the three approach, on both Intra and Inter. 



> I was surprised that "split" did not work better, and Real's and 

> "interleave" works good even though they break the original correlation 

> on the scan thus we assumed they are less efficient. 

> 

> I think, one of the reason on this is as follows; Distribution of the 

> coefficients on the scanned sequence of coefficients does have a property 

> of (simply) concentrating at low frequency, but is relatively random on 

> each part of the sequence. Each part of the sequence just have the same 

> tendency of distribution, but does not have special dependency on, for 

> example, frequency band of that part. Thus a way of extraction 

> coefficients, from the sequence of coefficients does not bather (i.e., 

> does not make ineffective) the CAVLC design. Rather, "split" provides 

> unusual concentration/depopulation of the coefficients and bather the 

> CAVLC design. 
> 

> If this is the case, "interleave" should be the best approach to 

> introduce CAVLC on ABT. It can apply to interlace scan ("field scan") and 

> it should work good on the scan. Since it does not divide the 

> coefficients into low/high frequency quadrant, the divided blocks of 

> coefficients would not have particular correlation, thus N (NumCoeff 

> prediction) should equally work well on each block without special rule 
for ABT. 

> 

> I think we can patent this. A question I have is; should we refer to 

> Real's approach as an prior art? 

> 

> Please comment and give us another point of view. 
> 

> Thanks and best regards, 

> Satoru Adachi 
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Exhibit E 



CERTIFICATION OF TRANSLATION 



I, the undersigned, hereby declare that: 

I am knowledgeable in both English and Japanese languages, and that I believe 
that the Japanese translation attached to this certification is a true and accurate 
translation of the Japanese document attached hereto and entitled "REQUEST 
FOR FILING OF PATENT APPLICATION." 

I hereby declare that all statements made herein of my own knowledge are true 
and that all statements made on information and belief are believed to be true; and 
further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, 
under Section 1001 of Title 18 of the United States Code. 




Name: ^fo-S^ HoN*. 



Signature 




X-Mailer: QUALCOMM Windows Eudora Version 4. 3. 2- J 
Date : 

To: spg-visual 

From: Satoshi Adachi <adachi> 

Content -Type: multipart /mixed: 

boundary="===============_18484329==_" 

Subject: [epg-visual 267] Reverse Prediction Delay Notification 
Draft Patent Application 

"How are you" from Adachi . 

I am currently working on drafting patent applications for 
proposals to the next JVT meeting. I just completed drafting of 
one patent application and am sending it to you. I apologize for 
not advising you in advance, but I have already sent the draft 
patent application to a patent firm via our Intellectual Property 
Department because the due date for the proposals (midnight on the 
4th) is closing in. 

• Notifying and defining method used to adjust a decoded image 
output delay associated with a reverse prediction 

In addition to this patent application, I will complete drafting 
of the following two patent applications as soon as I can and 
expect to hopefully have the applications filed this week. 

• An output frame buffer managing method for a reverse prediction 
using multiple reference frames 

■ A method for applying context-adaptive variable-length coding to 
adaptive orthogonal transform size image encoding 



Your comments would be appreciated. 
Attachment: Patent Application.zip 



X-Hai ler: QUALCQMM Windows Eudora Ver sion 4. 3. 2- J 

Date : VHHHMHHI 
To: spg-visua! 

Frooj: $it tf <adachi> ^ _ 

Content-Type: mult i part/mi xed : boundary="= == =: 

Subject: [spg-visual 267] gfiffl^iMafiftilfl) «RIWtt»* 



• a*is^ai=»it4«#ii«Hi*aiiwaa)fcar)a)att. ««*a 





Exhibit F 



Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG 
(ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6) 

5 th Meeting: Geneva, Switzerland, 9-17 Oct, 2002 



Document: JVT-E120 
Filename: JVT-E120.doc 



Title: 



CAVLC Cleanup to Accommodate ABT including Field Scans 



Status: Input Document to JVT 

Purpose: Proposal 

Author(s) or S.Adachi, S.Kato, T.K.Tan, and 

Contact(s): M.Etoh 



Tel: +65-6482-5493 

Email: tktan@spg. yrp.nttdocomo.co.jp 



3-5, Hikari-no-oka, Yokosuka, Japan 



Source: 



NTT DoCoMo, Inc. 



1 Introduction 

In this document we propose to harmonize the CAVLC entropy decoding with the dedicated 
VLC decoding of the ABT decoding process. 

CAVLC was designed to work on sub blocks of 4x4 coefficients, whereas ABT allows block 
sizes of 8x8, 8x4, 4x8 and 4x4. 

A simple harmonization process where the block sizes of 8x8, 8x4 and 4x8 are subdivided into 
blocks of 4x4 is proposed. This would allow all entropy decoding to be done using the same 
CAVLC method without any modification. 

2 Proposed Method. 

2. 1 Entropy coding 

In this method we proposed that the scanning of the coefficients of the 8x8, 8x4, 4x8 and 4x4 be 
done as currently described in Sections 8.6.1 and 12.4.2. No changes are proposed to the 
scans be it the zig-zag scan or the field scan. The proposed method works equally well with all 
existing scans. 

In the case of 8x8 block size, the resulting 64 scanned coefficients numbered from 0, 1, 2, 3, 
...to 63 are redistributed into 4 groups where the first group comprises of 16 scanned 
coefficients numbered 0, 4, 8, ... to 60. The second group comprises of 16 scanned 
coefficients numbered 1, 5, 9, ... to 61. The third group comprises of 16 scanned coefficients 
numbered 2, 6, 10, ... to 62, and the fourth group comprises of 16 scanned coefficients 
numbered 3, 7, 11, ... to 63. 

In the case of 4x8 and 8x4 block sizes, the resulting 32 scanned coefficients numbered from 0, 
1, 2, 3, ... to 31 are redistributed into 2 groups where the first group comprises of 16 scanned 
coefficients numbered 0, 2, 4, ... to 30. The second group comprises of 16 scanned 
coefficients numbered 1, 3, 5, ... to 31. 

No redistribution is needed for the case of 4x4 blocks. 

All the groups of 16 scanned coefficients are then CAVLC decoded as described in section 
9.1.6. 
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2.2 Table Selection: 



For the purpose of determining N for the table selection the Groups are assumed to have the 
following physical locations 





Group 1 


Group 2 


Group 3 


Group 4 



a) 8x4 blocks 



b) 4x8 blocks 



c) 8x8 blocks 



3 Results 

The experiment was conducted comparing the proposed solution to the performance of the 
JM4.0d implementation where the ABT blocks are coded using the dedicated VCL coding as 
described in Section 12.5. 

For comparison purposes the results posted by RealNetworks on the reflector were also 
included. 

The result shows that the proposed solution always perform better that the JM4.0d. For Inter 
coding a maximum improvement of up to 14.14% was observed with and average of 11.17% 
over all sequences. For Intra coding a maximum improvement of 40.03% was observed with 
and average of 30.50% over all sequence. 



Inter 






Real Networks 

MAX MIN AVE 


DoCoMo 

MAX MIN AVE 


QCIF 


Foreman 
Container 
Silent 
News 


8.18 -2.05 3.23 
8.72 3.26 5.62 
10.77 -3.65 2.94 
11.60 1.71 5.69 


9.00 1 10 4.86 
9.47 3.35 6.39 
10.26 -0.82 3.98 
12.42 3.87 7.08 


CIF 


Mobile 
Paris 
Tern pete 
Bus 

Flowergarden 


11.51 2.14 5.96 
9.67 3.07 5.24 
10.93 0.47 4.47 
9.58 3.82 6.44 
13.80 0.68 7.05 


12.17 3.41 7.18 
10.31 4.48 6.37 
11.99 3.19 6.01 
10.75 6.37 7.99 
14.14 2.24 8.02 




All 


10.53 1.05 5.18 


11.17 3.02 6.43 
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Intra 






Real Networks 




DoCoMo 










MAX 


MIN 


AVE 


MAX 


MIN 


AVE 




Foreman 


23.03 


-1.34 


9.32 


22.55 


1.34 


9.71 




Container 


27.96 


4.34 


14.29 


27.90 


6.01 


14.59 




Silent 


25.52 


0.16 


9.26 


25.22 


1.97 


9.56 


QCIF 


Mews 


29.78 


4.73 


15.47 


29.63 


3.35 


15.12 




Mobile 


37.90 


5.06 


17.79 


37.58 


5.74 


18.05 




Paris 


30.66 


5.44 


15.24 


30.32 


6.44 


15.31 




Tempete 


33.30 


1.15 


13.54 


33.23 


2.82 


13.86 




Bus 


28.60 


4.41 


12.58 


28.06 


4.72 


12.68 


CIF 


Flowergarden 


40.24 


2.19 


18.32 


40.03 


4.69 


19.34 




All 


30.78 


2.91 


13.98 


30.50 


4.12 


14.25 



Similarly the performance of the proposed solution also improves for the cases involving 
interlace material and field scans. 



4 Conclusions 

We propose to harmonize the VLC coding into a single method. There is no additional cost in 
complexity apart for reorganizing the scanned coefficients and the entropy decoding is 
simplified. 

The simplification also results in improved coding efficiency of up to 14.14 % in inter coding and 
40.03% in intra coding. 

The proposed text is provided below. 



5 Proposed Text: 



5.1 Modification 1: 

Merge Subclause 8.6. 1 Zig Zag Scan with Subclause 12.4.2 
Replace first paragraph (including Figure 8-12) and last paragraph 
With 

Subclause 12.4.2 

While retaining 2 nd and 3 rd paragraphs 



5.2 Modification 2: 

Replace the last two paragraphs of subclause 9. 1.6 

Zig-zag scanning as described in subclause 9.4.1 is used, but in the decoding of coefficient data, both levels and 
runs, the scanning is done in reverse order. Therefore the signs of Tls are decoded first (in reverse order), then the 
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Level information of the last coefficient in the zig-zag scan order not included in the Tls, and so on. Run 
information is decoded similarly. First Total number of zeros in Runs is decoded, followed by Run before the last 
nonzero coefficient in the zig-zag scan order, and so on. 

If adaptive_block_size Jransform flag = = 1 , the VLC method for decoding 4x4, 4x8, 8x4, and 8x8 luma 
coefficient blocks is specified in subclause 12.5.1. 

With 

In decoding of coefficient data, both levels and runs, the scanning is done in reverse order. Therefore the signs of 
Tls are decoded first (in reverse order), then the Level information of the last coefficient in the zig-zag scan order 
not included in the Tls, and so on. Run information is decoded similarly. First Total number of zeros in Runs is 
decoded, followed by Run before the last nonzero coefficient in the zig-zag scan order, and so on. 

The above decoding operation results in groups of 16 zig-zag scanned coefficients. 

In the case of 8x8 block size, four groups of 16 scanned coefficients are decoded in the above manner. These four 
groups of scanned coefficients are then combined in an interleave manner taking one coefficient from each group 
iteratively, starting with the first group, to form a single array of 64 scanned coefficients. 

In the case of 4x8 and 8x4 block sizes, two groups of 16 scanned coefficients are decoded in the above manner. 
These two groups of scanned coefficients are then combined in an interleave manner taking one coefficient from 
each group iteratively, starting with the first group, to form a single array of 32 scanned coefficients. 

No interleaving is necessary for the case of 4x4 block size. 

The arrays of 16, 32 and 64 scanned coefficients are then inverse zig-zag scanned according to the zig-zag and field 
scans as described in subclause 8.6.1. 



5.3 Modification 3: 

Add to subclause 9. 1. 6.2 Table selection before the Tale 9-8. 

For the purpose of determining N for the table selection the groups of 16 scanned coefficients decoded in subclause 
9.1 .6 are assumed to have the following physical locations 





Group 1 


Group 2 


Group 3 


Group 4 



a) 8x4 blocks 



b) 4x8 blocks 



c) 8x8 blocks 
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/Date stamp: 10/8/20021 
14-0422 

[Title of the Invention] 

IMAGE ENCODING METHOD, IMAGE DECODING METHOD, IMAGE ENCODING 
APPARATUS, IMAGE DECODING APPARATUS, IMAGE TRANSMISSION SYSTEM, 
IMAGE STORING SYSTEM, AND IMAGE REPRODUCTION SYSTEM, CAPABLE OF 
IMPLEMENTING EFFICIENT ENTROPY CODING OF ORTHOGONAL TRANSFORM 
COEFFICIENTS, IN AN ORTHOGONAL TRANSFORM PERMUTING SELECTION 
AMONG MULTIPLE BLOCK SIZES 

[Claims] 

<Encoding Method Claim - Basic Claim> 
[Claim 1] 

An image encoding method of dividing image signals into blocks, performing an 
orthogonal transform, sequentially reading resultant orthogonal transform coefficients to obtain a 
coefficient string, and performing entropy coding thereon, the image encoding method 
comprising: 

a means for selecting a size of a block for the orthogonal transform, out of a plurality of 
blocks of different sizes; 

a means for performing entropy coding adapted to the coefficient string in a block of the 
minimum size from among the plurality of blocks; 

a means for dividing, when a block of a larger size is selected, the coefficient string in 
the block into a plurality of coefficient strings formed from a number of coefficients equal to a 
number of coefficients of a coefficient string in a block of the minimum size,; and 

a means for entropy encoding of the divided coefficient string. 

<Encoding Method Claim, Coefficient Dividing Method, Alternating Readout> 
[Claim ] 

An image encoding method according to claim 1, wherein the means for dividing the 
coefficient string is configured to sequentially read the coefficients of the coefficient string from 
the low frequency region and to alternately assign them to the plurality of coefficient strings 
formed from a number of coefficients equal to the number of coefficients of a coefficient string 
in a block of the minimum size, thereby obtaining the divided coefficient strings. 



<Encoding Method Claim, Coefficient Dividing Method, For Each Frequency Band> 
[Claim ] 
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An image encoding method according to claim 1, wherein the means for dividing the 
coefficient string is configured to sequentially read the coefficients of the coefficient string from 
the low frequency region, to read only the number of coefficients equal to the number of 
coefficients of a coefficient string in a block of minimum size, and lo repeatedly perform this to 
obtain divided coefficient strings. 

<Decoding Method Claim - Basic Claim> 
[Claim ] 

An image decoding method of decoding data encoded by dividing image signals into 
blocks, performing an orthogonal transform, sequentially reading resultant orthogonal transform 
coefficients to obtain a coefficient string, and performing entropy coding thereof, the image 
decoding method comprising: 

a means for selecting a size of a block for the orthogonal transform, out of a plurality of 

blocks of different sizes; 

a means for decoding data encoded by entropy coding adapted to the coefficient string in 
a block of the minimum size from among the plurality of blocks; 

a means for constructing when a block of a larger size is selected, the coefficient string 
with the same number of coefficients included in the block from a plurality of coefficient strings 
formed from a number of coefficients equal to a number of coefficients of a coefficient string in 
a block of the minimum size ; and 

a means for decoding the plurality of coefficient strings from the encoded data. 

<Decoding Method Claim, Coefficient Dividing Method, Alternating Readout> 
IClaim j 

An image decoding method according to claim 1 (sic), wherein the means for 
constructing the coefficient string is configured to read, from the low frequency regions, the 
respective coefficients alternately from the plurality of coefficient strings each formed from a 
number of coefficients equal to the number of coefficients of a coefficient string in a block of the 
minimum size, writing them into a new coefficient string, thereby obtaining the constructed 
coefficient string. 

<Decoding Method Claim, Coefficient Dividing Method, For Each Frequency Band> 
[Claim ] 

An image decoding method according to claim 1 (sic), wherein the means for 
constructing the coefficient string is configured to read all of the coefficients of a single 
coefficient string from the low frequency region, from the plurality of coefficient strings formed 
from a number of coefficients equal to the number of coefficients of a coefficient string in a 
block of the minimum size, repeatedly writing them into a new coefficient string, thereby 
obtaining the constructed coefficient strings. 
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<Apparatus Claim> 
[Claim ] 
<System Claim> 
[Claim ] 

An image processing system formed from an image encoding apparatus and an image 
decoding apparatus, wherein the encoding apparatus comprises an image encoding apparatus of 
any one of claims to ; and an image decoding apparatus of any one of claims to . 

<Computer Encoding Program> 

[Claim ] 

An image encoding program for executing the steps of the image encoding method 
according to any of claims to , in a computer used as an image encoding apparatus. 

<Computer Decoding Program> 

[Claim ] 

An image decoding program for executing the steps of the image decoding method 
according to any of claims to , in a computer used as an image decoding apparatus. 
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[Detailed Description of the Invention] 
[Technical Field] 

The present invention relates to an image encoding method, an image decoding method, 
an image encoding apparatus, an image decoding apparatus, an image transmission system, an 
image storing system, and an image reproduction system. 

[Prior Art] 

Encoding techniques of image signals are used for transmission and for storing and 
reproducing image signals of still images, moving images, and so on. Such techniques include 
known international standard encoding methods, e.g., ISO/IEC International Standard 10918 
(hereinafter referred to as JPEG) as an encoding technique for still images, and ISO/IEC 
International Standard 14496-2 (MPEG-4 Visual, which will be referred to hereinafter as MPEG- 
4) as an encoding technique for moving images. A newer known encoding method is ITU-T 
Recommendation H.264; ISO/IEC International Standard 14496-10 (Joint Final Committee Draft 
of Joint Video Specification, f^//ftp.imtc-files.org/jvt-experts/2002_07_KJagenfur^ 
D157.zip, which will be referred to hereinafter as H.26L), which is a video encoding method 
intended for joint international standardization by ITU-T and ISO/IEC. An example of a 
reference pertaining to typical encoding techniques used in these image encoding methods is 
Kokusai hyojun gazofugoka no kiso gijutsu [Basic Technologies of International Image Coding 
Standards] by Fumitaka Ono and Hiroshi Watanabe, published March 20, 1998 by Corona 
Publishing Co., Ltd. 

Image signals demonstrate close correlations between spatially neighboring pixels and 
thus transformation into the frequency domain leads to deviation of information to the low 
frequency region, which enables reduction of redundancy by making use of the deviation. 
Therefore, the typical image encoding methods adopt a technique of subjecting image signals to 
an orthogonal transform to transform them into orthogonal transform coefficients in the 
frequency domain, so as to achieve deviation of signal components to the low frequency region. 
Furthermore, the coefficient values are quantified so that small-valued coefficients are converted 
to zero-valued coefficients. A coefficient string is made by sequentially reading the coefficients 
from the coefficients in the low frequency region and is subjected to entropy coding to take 
advantage of the coefficient values, thus achieving efficient encoding with reduction of 
redundancy. 

In this case, the Discrete Cosine Transform (DCT) is commonly used in terms of 
encoding efficiency and ease of implementation. The orthogonal transform such as the DCT is 
carried out in units of blocks resulting from division of image signals into blocks each consisting 
of a plurality of pixels. The size of the blocks, as well as the property of the image signals, 
largely affects the encoding efficiency. 

When image signals demonstrate only small change in the spatial property, image signals 
to be transformed into orthogonal transform coefficients in a similar frequency region are widely 
distributed on an image, and the redundancy can be reduced more with increase in the size of the 
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blocks, i.e., the size of the orthogonal transform, so as to increase the encoding efficiency, as 
compared with using smaller blocks, which raise the need for repeatedly expressing identical 
orthogonal transform coefficients. When image signals demonstrate large change in the spatial 
property on the other hand, the increase in the size of the blocks results in obtaining orthogonal 
transform coefficients havingvarious frequency components, thus decreasing the deviation of 
coefficients, which makes efficient entropy coding difficult and thus decreases the encoding 
efficiency. 

In order to take advantage of the change of encoding efficiency due to the changes in the 
sizes of the blocks for the orthogonal transform and the property of image signals, the technology 
utilized is one of preparing orthogonal transform means adapted for a plurality of block sizes in 
advance and adaptively selecting and using a size achieving the best encoding efficiency out of 
them. This technology is called Adaptive Block size Transforms (ABT) and is adopted in H.26L. 
FIG. 1 shows orthogonal transform blocks used for ABT in H.26L. The ABT permits a size 
achieving the best encoding to be selected out of four types of orthogonal transform block sizes 
shown in FIGS. 1 (b) - 1 (e), for each macroblock of 16 x 16 pixels shown in FIG. 1 (a). Pixel 
values of the macroblocks are equally divided in blocks, according to the selected size and are 
then subjected to the orthogonal transform. By implementing such selection, it becomes feasible 
to achieve efficient reduction of redundancy through the use of the orthogonal transform in 
accordance with the change in the spatial property of image signals in the macroblocks. 
Reference should be made to H.26L as to more specific details of the ABT. 

The entropy coding for the orthogonal transform coefficients obtained by the orthogonal 
transform is effected on a coefficient string obtained by sequentially reading the orthogonal 
transform coefficients from the coefficients in the low frequency region. FIG 2 (a) shows an 
order of reading coefficients in an orthogonal transform block of 4 x 4 pixels. Since the 
coefficients obtained by the orthogonal transform are arranged with the lowest frequency 
component (i.e., the direct current component) at the upper left corner, the coefficients are read 
out in order from the upper left coefficients to obtain a coefficient string consisting of sixteen 
coefficients as shown in FIG. 2 (b). This reading order is called zig-zag scan. 

The coefficients obtained by the orthogonal transform are noncorrelated with each other, 
and the signal components deviate to the low frequency region. For this reason, when they are 
further quantized, the lower frequency coefficients are more likely to be nonzero coefficient 
values, so that many zero-valued coefficients appear in the coefficient string. For example, it 
produces a sequence of coefficient values as shown in FIG. 2 (c). Therefore, for efficient 
entropy coding of the coefficient string of this distribution, it is common practice in encoding of 
images to perform the encoding by expressing the coefficient string by the numbers of 
continuous zero coefficients preceding a nonzero coefficient (runs) and coefficient values 
(levels) of the nonzero coefficients. Such encoding with runs and levels is also used in the 
entropy coding of orthogonal transform coefficients by the ABT. 

On the other hand, in order to increase the efficiency more in the entropy coding as 
described above, H.26L employs a technology called Context-based Adaptive Variable Length 
Code (CAVLC), which is applied to the orthogonal transform without the use of ABT, i.e., to 
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cases where the orthogonal transform is always carried out in units of orthogonal transform 
blocks of 4 x 4 pixels. 

The CAVLC in II.26L utilizes the following features: the maximum number of 
coefficients in the coefficient string obtained from the orthogonal transform blocks of 4 x 4 
pixels is 16; the magnitude of runs is restricted by this maximum number; and the magnitude of 
levels tends to be larger at lower frequencies. A large number of encoding tables used in 
variable length encoding are prepared as optimized tables for respective conditions, and they are 
applied while being sequentially switched, so as to increase the encoding efficiency. 

For example, in the case where runs are encoded in order, the first run can take a variety 
of values from 0 to 14 (according to the definition of runs in H.26L, the maximum number of 
runs is 14, which is only two smaller than the total number of coefficients). On the other hand, 
runs appearing in the last stage of the sequential coding of runs can take only limited run values, 
because there is an upper limit to the number of coefficients in the coefficient string. 
Accordingly, as shown in FIG. 3, the right-side encoding table with the largest number of 
elements of the encoding table is applied to the runs appearing in the initial stage, and the left- 
side encoding tables with the smaller number of elements of the encoding table are applied to 
runs appearing in the last stage. This permits assignment of codes of smaller bit counts and thus 
implements efficient entropy coding. The CAVLC achieves the efficient encoding by making 
use of the conditions such as the maximum number of coefficients in the blocks, and by placing 
restrictions on the range where values to be encoded can take. Reference should be made to 
H.26L as to more specific details of the CAVLC. 

[Problems to be Solved by the Invention] 

By applying the foregoing CAVLC to the ABT, it can be expected that more efficient 
entropy coding will also be achieved in the coefficient strings of the ABT. 

However, the CAVLC achieves the increase in encoding efficiency by optimizing the 
encoding tables used in variable length coding for the respective conditions, based on the 
maximum number of coefficients in the blocks, and applying the encoding tables to the 
switching while switching among them. 

When ABT is used, the number of coefficients differ in each of the blocks which differ in 
size: 64 in the case of 8 x 8 blocks in FIG. 1 (b), 32 in the case of 8 x 4 and 4 x 8 blocks in FIGS. 
1 (c) and 1 (d), and 16 in the case of 4 x 4 blocks in FIG. 1 (e). For this reason, the application 
of the CAVLC requires consideration be given to the huge number of conditions that can occur 
in the respective cases. 

For example, supposing the encoding tables are set according to the maximum number of 
coefficients in coefficient strings, like the encoding tables of runs shown in FIG. 3, a huge 
number of encoding tables must be prepared: in the case of 8 x 8 blocks with the number of 
coefficients being 64 7 it is necessary to prepare the encoding tables ranging from the encoding 
table with the number of elements being 2 to the encoding table with the number of elements 
being 62. Likewise, in the cases of 8 x 4 and 4 x 8 blocks with the number of coefficients being 
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32, the encoding tables must be prepared from that of the number of elements being 2 to that of 
the number of elements being 30. 

When it was attempted to apply the entropy coding optimized for characteristics of 
coefficients like the CAVLC to the orthogonal transforms selectively using the orthogonal 
transform blocks of different sizes like the ABT as described above, there was the problem that 
the number of encoding tables to be prepared became huge, and the memory capacity necessary 
for retention of the encoding tables became large. Since it also involved the use of different 
encoding tables for blocks of the respective sizes, as well as different selection procedures 
thereof, there was the problem that the procedure in the entropy coding became complicated, and 
thus the implementing means and the apparatus structure became complicated. 

The present invention has been accomplished in order to solve the above problems, and 
an object of the present invention is to provide an image encoding method, an image decoding 
method, an image encoding apparatus, an image decoding apparatus, an image transmission 
system, an image sotring system, and an image reproduction system enabling efficient entropy 
coding in the orthogonal transform of variable sizes. 

[Means for Solving These Problems] 

<Encoding Method Claim - Basic Claim> 

In order to achieve the above object, the present invention provides an image encoding 
method of dividing image signals into blocks, performing an orthogonal transform, sequentially 
reading resultant orthogonal transform coefficients to obtain a coefficient string, and performing 
entropy coding thereon, the image encoding method comprising: 

a means for selecting a size of a block for the orthogonal transform, out of a plurality of 
blocks of different sizes; 

a means for performing entropy coding adapted to the coefficient string in a block of the 
minimum size from among the plurality of blocks; 

a means for dividing, when a block of a larger size is selected, the coefficient string in 
the block into a plurality of coefficient strings formed from a number of coefficients equal to a 
number of coefficients of a coefficient string in a block of the minimum size,; and 

a means for entropy encoding of the divided coefficient string. 

Accordingly, in the image encoding method according to the present invention, when a 
block of a larger size is selected to be subjected to the orthogonal transform, the coefficient 
string in that block is first divided into coefficient strings and then the entropy coding is 
performed. This permits the entropy coding adapted to the coefficient string in the block of the 
minimum size to be applied in entropy coding of the coefficient string, whereby it is feasible to 
implement efficient entropy coding of orthogonal transform coefficients, without complicating 
the procedure of entropy coding. 

^Encoding Method Claim, Coefficient Dividing Method, Alternating Readout> 
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The means for dividing the coefficient string is configured to sequentially read the 
coefficients of the coefficient string from the low frequency region and to alternately assign them 
to the plurality of coefficient strings formed from a number of coefficients equal to the number 
of coefficients of a coefficient string in a block of the minimum size, thereby obtaining the 
divided coefficient strings. 

<Encoding Method Claim, Coefficient String Dividing Method, For Each Frequency Band> 

The means for dividing the coefficient string is configured to sequentially read the 
coefficients of the coefficient string from the low frequency region, to read only the number of 
coefficients equal to the number of coefficients of a coefficient string in a block of minimum size, 
and to repeatedly perform this to obtain divided coefficient strings. 

<Decoding Method Claim - Basic Claim> 

Tthe present invention also provides an image decoding method of decoding data 
encoded by dividing image signals into blocks, performing an orthogonal transform, sequentially 
reading resultant orthogonal transform coefficients to obtain a coefficient string, and performing 
entropy coding thereof, the image decoding method comprising: 

a means for selecting a size of a block for the orthogonal transform, out of a plurality of 
blocks of different sizes; 

a means for decoding data encoded by entropy coding adapted to the coefficient string in 
a block of the minimum size from among the plurality of blocks; 

a means for constructing when a block of a larger size is selected, the coefficient string 
with the same number of coefficients included in the block from a plurality of coefficient strings 
formed from a number of coefficients equal to a number of coefficients of a coefficient string in 
a block of the minimum size ; and 

a means for decoding the plurality of coefficient strings from the encoded data. 

Accordingly, in the image decoding method according to the present invention, when a 
block of a larger size is selected to implement decoding of the encoded data subjected to the 
orthogonal transform, the coefficient string in that block is constructed from coefficient strings 
comparable to that block. This permits the decoding of coefficient strings from data encoded by 
using entropy coding adapted to the coefficient string in the block of the minimum size, whereby 
it is feasible to implement efficient entropy coding of orthogonal transform coefficients, without 
complicating the procedure of entropy coding. 

<Decoding Method Claim, Coefficient Dividing Method, Alternating Readout> 

The means for constructing the coefficient string is configured to read, from the low 
frequency regions, the respective coefficients alternately from the plurality of coefficient Strings 
each formed from a number of coefficients equal to the number of coefficients of a coefficient 
string in a block of the minimum size, writing them into a new coefficient string, thereby 
obtaining the constructed coefficient string. 

<Decoding Method Claim, Coefficient Dividing Method, For Each Frequency Band> 
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The means for constructing the coefficient string is configured to read all of the 
coefficients of a single coefficient string from the low frequency region, from the plurality of 
coefficient strings formed from a number of coefficients equal to the number of coefficients of a 
coefficient string in a block of the minimum size, repeatedly writing them into a new coefficient 
string, thereby obtaining the constructed coefficient strings. 

[Embodiments of the Invention] 

<First Embodiment 

The preferred embodiments of the image encoding method, image decoding method, 
image encoding apparatus, image decoding apparatus, image transmission system, image storing 
system, and image reproduction system according to the present invention will be described in 
detail below with reference to the drawings. 

The description will be based on the premise that the encoding and decoding in the 
description hereinafter are implemented on the basis of H.26L, and the part without specific 
description about the operation in the image encoding is supposed to conform to the operation of 
H.26L. However, the present invention is not limited to H.26L. 

An embodiment of the present invention will be described. In the encoding according to 
the present embodiment, orthogonal transform coefficients in blocks of the respective sizes of 
the ABT in H.26L are divided into a plurality of coefficient strings consisting of coefficients in a 
number equal to the number of coefficients in a coefficient string of a 4 x 4 block. This makes it 
feasible to perform the entropy coding by the CAVLC of H.26L defined so as to be adapted to 
the 4 x 4 blocks. 

It is assumed that in the encoding, first, the ABT in H.26L is applied to one macroblock, 
a size achieving the best encoding efficiency is selected out of the blocks shown in FIGS. 1 (b)- 
(e), and the orthogonal transform is effected in units of blocks of the selected size. 

It is also assumed that the CAVLC in H.26L is employed in the entropy coding of 
orthogonal transform coefficients. Namely, it is assumed that only the variable length encoding 
adapted to the encoding of orthogonal transform coefficients for the 4 x 4 block shown in FIG. 1 
(e) is defined. 

For example, let us suppose herein that the 8 x 8 block in FIG. 1 (b) is selected. The 
following readout operation of reading the orthogonal transform coefficients is carried out for 
this 8 x 8 block. First, 64 coefficients in the 8 x 8 block are read out by zig-zag scan as shown in 
FIG. 4 (a), to obtain a coefficient string as shown in FIG. 4 (b). 

Then, this coefficient string is divided into four coefficient strings each consisting of 
sixteen coefficients, the number of which is the same as the number of coefficients in the 
coefficient string of the 4 x 4 block. Here the coefficients in the original coefficient string are 
read out from the low frequency region and alternately assigned respectively to the four 
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coefficient strings, thereby obtaining the divided coefficient strings. FIGS. 4 (c) and 4 (d) show 
this readout operation. Since the coefficients are alternately assigned to the respective 
coefficient strings from the coefficients in the low frequency region, the coefficients read out in 
the order of the 0 th , 4 th , 8 th , 12 th . . . in ihe original coefficient siring are each read out and 
assigned to the first divided coefficient string, and the coefficients read out in order of the 1 st , 5* , 
9 th , 1 3 th ... in the original coefficient string are each read out and assigned to the second divided 
coefficient string. The third and fourth divided coefficient strings are not illustrated in FIG. 4. 

Similarly, when the 8 x 4 block or the 4 x 8 block of FIG. 1 (c) or FIG. 1 (d) is selected, 
32 coefficients are divided into two coefficient strings consisting of 16 coefficients. The readout 
method for obtaining the divided coefficient strings is also similar to that in the case of the 8 x 8 
block, except that the number of coefficient strings alternately assigned is 2 instead of 4. The 
coefficients in the original coefficient string are read out from the low frequency region and 
alternately assigned to the two coefficient strings. 

The coefficient strings obtained in this way are entropy encoded according to entirely the 
same procedure as the encoding of CAVLC when ABT is not used, and outputted in order as ' 
encoded data of orthogonal transform coefficients in the ART block. 

At this time, the CAVLC of H.26L utilizes the space context to switch the applied 
encoding table on the basis of the number of nonzero coefficients in an adjacent 4x4 block. For 
this reason, arrangements of coefficient strings in the ABT blocks larger than the 4 x 4 block 
after division are defined. The definitions are shown in FIG. 5. For example, the 8 x 8 block 
shown in FIG, 5 (a) is handled such that the first divided coefficient string illustrated in FIG, 4 

(c) is located at the position of "1" and the second divided coefficient string illustrated in FIG, 4 

(d) is located at the position of "2." Using the definitions of this arrangement, the space context 
for the divided coefficient strings in the ABT blocks, or the space context for the 4 x 4 blocks 
adjacent to the ABT blocks is handled as in the technique of the CAVLC in H.26L without any 
change. 

In the decoding, the original orthogonal transform matrix can be obtained according to a 
procedure reverse to the procedure in encoding. 

Let us suppose that the ABT in H.26L is applied to one macroblock, a size is designated 
out of the blocks shown in FIG. 1 (b) - (e), and encoded data for this macroblock results from 
the orthogonal transform carried out in units of the ABT block. 

At this time the encoded data contains encoded data obtained by entropy coding of the 
divided coefficient strings by the CAVLC, in order as encoded data of orthogonal transform 
coefficients in the ABT blocks. Accordingly, it is sequentially decoded according to the 
procedure of the CAVLC to obtain the divided coefficient strings. 

Since these divided coefficient strings are coefficient strings divided by the readout 
method shown in FIG. 4, the original orthogonal transform coefficient block can be obtained by 
conversely writing the coefficients of the divided coefficient strings into the original coefficient 
string and farther writing the resultant coefficient strings into the orthogonal transform 
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coefficient block. The procedure thereafter is the same as the decoding procedure with 
application of the ABT in H.26L. 

In the present embodiment, the zig-zag scan was applied lo readout of orthogonal 
transform coefficients, but the readout method of coefficients in application of the present 
invention does not have to be limited to the zig-zag scan. For example, the present invention 
may also be applied to cases of application of a field scan for field encoding in interlaced images, 
which is defined in the ABT of H.26L. In this application, the dividing technique of the 
coefficient strings in the present embodiment can be applied as it is. 

The present embodiment showed the readout method for obtaining the coefficient strings 
after division as the alternating readout shown in FIG. 4, but it is also possible to obtain the 
divided coefficient strings by another readout method different therefrom. For example, as 
shown in FIGS. 6 (c) and 6 (d), sixteen consecutive coefficients are each read out from the 
original coefficient string from the low frequency region and each of them may be assigned to 
one of the divided coefficient strings. 

In the present embodiment, the readout of orthogonal transform coefficients in the 
encoding is implemented so as to perform a first readout for obtaining the coefficient string from 
the orthogonal transform block, and then to perform a second readout for obtaining a plurality of 
divided coefficient strings. The writing of orthogonal transform coefficients in the decoding is 
implemented so as to perform a first writing for obtaining the constructed coefficient strings, and 
then to perform a second writing for obtaining the orthogonal transform block. However, the 
readout and writing of coefficients according to the present invention do not have to be limited to 
these methods, but may also be implemented by a variety of readout and writing methods that 
can obtain coefficient strings in the desired arrangement. For example, it is also possible to 
implement such readout as to immediately obtain a plurality of divided coefficient strings in the 
first coefficient readout from the orthogonal transform block. The writing from the divided 
coefficient strings may also be performed so as to immediately obtain the orthogonal transform 
block in the first coefficient writing. 

In the present embodiment, the divided coefficient strings were arranged as shown in FIG. 
5, and the space context from the adjacent 4x4 blocks is assumed to be handled by the CAVLC 
in H.26L without any change. However, at this time, the coefficient strings divided from the 
coefficient strings of the ABT blocks larger than the 4 x 4 blocks can be considered as having 
properties essentially different from the coefficient strings in the case of the original 4x4 block, 
and weight may be given to the numerical values used as the space context. Specifically, the 
number of nonzero coefficients can be used as the space context from the adjacent blocks, but 
the number of nonzero coefficients for the divided coefficient strings obtained from ABT blocks 
larger than the 4 x 4 blocks may be added with or multiplied by a constant when used as the 
space coefficient. Alternatively, when divided coefficient strings are obtained by continuously 
reading out the coefficients from the low frequency region as shown in FIG. 6, different 
constants may be added to or multiplied by coefficients read out from the low frequency region 
and coefficients read out from the high frequency region. 
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The description of the present embodiment was implemented on the basis of H.26L, and 
the description was based on the ABT and CAVLC in the H.26L. However, the image encoding 
methods to which the present invention can be applied are not limited to H.26L, and it is possible 
to apply the present invention to a variety of image encoding methods permitting selection from 
a plurality of block sizes for the orthogonal transform and using entropy coding adapted to the 
orthogonal transform coefficients. 

[Advantageous Effects of the Invention] 

The image encoding method, image decoding method, image encoding apparatus, image 
decoding apparatus, image transmission system, image storing system, and image reproduction 
system according to the present invention, as described in detail above, yield the following 
advantageous effects. Efficient entropy coding can be achieved without increasing the number 
of encoding tables in entropy coding, and without complicating the encoding tables and their 
selection procedure, by dividing the coefficient strings formed from resultant orthogonal 
transform coefficients into a plurality of coefficient strings equal in size to that of a coefficient 
string in a block of the minimum size, in cases where it is possible to select the size of the blocks 
for orthogonal transform from a plurality of blocks, and performing entropy coding adapted to 
the coefficient string in the block of the minimum size. 

[Brief Description of the Drawings] 

[FIG. 1] 

Diagram illustrating an orthogonal transform block used in Adaptive Block size 
Transforms (ABT) of H.26L. 

[FIG. 2] 

Diagram illustrating the readout method of coefficients in a 4 x 4 block, and an example 
of a coefficient string after readout. 

[FIG. 3] 

Diagram illustrating an encoding table of runs used in Context-based Adaptive Variable 
Length Code (CAVLC) of H.26L. 

[FIG. 4] 

Diagram illustrating an example of readout and dividing of orthogonal transform 
coefficients according to the present invention, performed on an 8 x 8 block. 

[FIG. 5] 

Diagram illustrating definitions of arrangements of coefficient strings after division 
according to the present invention, within the original blocks. 
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[FIG. 61 

Diagram illustrating an example of another method of readout and dividing of orthogonal 
transform coefficients according to the present invention, performed on an 8 x 8 block. 
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blocks for orthogonal transform is variable. 

6. Summary and Constitution 
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- Gist of claims 



Although encoding efficiency can be greatly improved by applying context-based 
adaptive variable length coding to orthogonal transform coefficients, it is necessary to employ 
encoding tables and encoding rules when such coding is to be applied in an image encoding 
method where the size of blocks subjected to orthogonal transform is variable, there was thus the 
problem that the means became complicated. The present invention applies context-based 
adaptive variable length coding only to blocks of a minimum size. Blocks of a larger size are 
divided into coefficient strings, so that the variable length coding is applied to a plurality of 
coefficient strings equal in size to that of a coefficient string in a block of the minimum size. 

COEFFICIENT MATRIX BEFORE READOUT 



m 



63 

COEFFICIENT STRING AFTER READOUT 



0 1 2 3 4 5 6 7 8 910.., 15 20 30 

□ I □ I i I 1 11 \ I I I II f 1 M 1 II 1 M I I I II I I I 1 



FIRST COEFFICIENT STRING 



0 4 8 12 16 20 24 28 32 

a i i m h m i i m q 18} p a i i m n m \ i a I ' 




0 1 2 3 ... 
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SECOND COEFFICIENT STRING 



1 5 9 13 17 21 25 29 33 

I H I i B m m n w i i i i m □ 50 naantai 




0 12 3... 16 



7. Advantageous Effects 

In accordance with the present invention, efficient encoding of orthogonal transform 
coefficients can be achieved without complicating the procedure and the structure, and 
enabling encoding by the same procedure when applied only to blocks of a minimum size, 
even when context-based adaptive variable length coding is applied in cases where the 
size of the blocks for orthogonal transform is variable. 

8. Remarks 
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